ON THE DEGENERATE PRINCIPAL SERIES OF 
COMPLEX SYMPLECTIC GROUPS 



PIERRE CLARE 

Abstract. We apply techniques introduced by Clerc, Kobayashi, 0rsted 
and Pevzner to study the degenerate principal series of Sp(n, C). An ex- 
plicit description of the K-types is provided and Knapp-Stein normalised 
operators are realised as symplectic Fourier transforms, and their K- 
spectrum explicitely computed. Reducibility phenomena are analysed 
in terms of if-types and eigenvalues of intertwining operators. We also 
construct a new model for these representations, in which Knapp-Stein 
intertwiners take an algebraic form. 



Introduction 

Background and purpose. Among the representations of reductive Lie 
groups, the so-called small representations have received a lot of attention 
for several years. After the accomplishments of the 90's, mainly obtained 
through algebraic methods (see the introduction and the references in |S]), 
recent progress has stemmed from the developement of new techniques in 
geometric analysis. Leading work in this direction was the series [21 EH 
111] , dealing with the minimal representation of general indefinite orthogonal 
groups. 

More recently, new techniques were introduced in this area by J.-L. Clerc, 
T. Kobayashi, B. 0rsted and M. Pevzner in relation with various problems 
in classical analysis [2] and representation theory |12) . The first of these 
works is devoted to the computation of certain triple integrals, and features 
the study of certain intertwining operators related to representations of the 
real symplectic group. More precisely, it is observed that some Knapp-Stein 
operators can be realised as Fourier transforms once normalised. This idea 
was already used in the work of A. Unterberger in the case of SL(2,R), see 
[16] . It was extended to SL(n,M) in |15] and to Sp(n,M) for the first time in 
|13j . It is taken further in [12J where, among many other results, a complete 
description of Knapp-Stein intertwiners between the degenerate principal 
series of Sp(n,M) is carried out. This point of view allows to establish very 
precise statements, such as K-type formulas and explicit computations of 
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the -RT-spectrum of intertwining operators. Another feature of |12| is the 
construction of what is named there the non-standard model for degenerate 
principal series. As a first application, this apparently new picture allows to 
define Knapp-Stein operators by a rather simple algebraic formula. 

The present article aims at adapting those techniques to the case of the 
complex symplectic group. Among the parabolically induced representations 
of Sp(n,C), those coming from maximal parabolic subgroups may be con- 
sidered as the most degenerate, since functions on the corresponding flag 
manifold depend on the smallest possible set of parameters. These represen- 
tations have first been investigated by K. I. Gross in [3]. 

Most of our arguments are rather directly inspired by the results in [2] 
and |12j . dealing with real symplectic groups. However, we tried to provide 
a self-contained and detailed presentation of these techniques while adapting 
them to the complex case. Moreover, the isotypic decomposition of the 
quaternionic orthogonal group action on square-integrable functions over the 
unit sphere, studied in Section [3TT| turns out to be slightly more delicate than 
the real and complex ones. It is also worth noticing Theorem [1] accounts 
more precisely for the reducibility phenomena than the original description 
in Gross' paper [3]. In particular, the characterisations by K-types and 
eigenspaces of the algebraic intertwiners in the non-standard model both 
seem new. 

Let us finally mention that the study of explicit Knapp-Stein intertwiners 
as geometric transforms and the computation of their X-spectrum such as 
the one provided in Proposition [3] are a current object of interest. Indeed, 
analogous results were recently obtained in [14] for special linear groups, 
using techniques of [JJ. 



Outline. The article is organised as follows: general notations are fixed and 
elementary facts regarding degenerate principal series of the complex sym- 
plectic groups are stated in Section [U In Section [21 we introduce certain 
Fourier transforms, establish some of their elementary properties and use 
them to normalise Knapp-Stein operators in Proposition [TJ In Section [3l we 
study the branching law of the degenerate principal series representations 
of Sp(n,C) with respect to the maximal compact subgroup K = Sp(n): we 
describe .fT-types in Proposition [2] and compute the eigenvalues of Knapp- 
Stein operators in Proposition [3l As a result, we are able to analyse the 
reducible elements in the degenerate principal series in terms of .fT-types and 
eigenspaces of te Knapp-Stein intertwiners in Theorem [TJ Finally, Section [4] 
is devoted to the description of the non-standard model of the degenerate 
principal series in the sense of |12| . The main result of this section is the 
computation of the normalised Knapp-Stein operators in this picture: The- 
orem [2] establishes that the intertwiners are defined by an algebraic formula 
in this setting. 
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1. Setting and notations 

1.1. The complex symplectic group. For any integer p > 1, let I p be the 

identity matrix of size p and let brackets denote the associate bilinear form 
on C p : 

v 

{X, Y) = ^ x kVk 

k=l 

for X = (xi, ...,x p ) and Y = (yi, . . . ,y p ). 

Throughout this article, n shall be a fixed positive integer and iV = 2n, so 
that a vector X in ~ C™ x C n can naturally be written X = (X 1 ,X 2 ). 
The complex symplectic form on C N is defined by 

u Jn (X,Y) = (X 2 ,Y 1 )-(X 1 ,Y 2 ), 

that is u n (X, Y) = (X, JY) where 

J 






-In ' 


In 






Abusing notations, we will usually drop the subscript indicating the dimen- 
sion and write uj = cu n when no confusion may result. 

By definition, the complex symplectic group is the group of complex in- 
vertible matrices preserving oj: 

Sp(n, C) = {g £ GL(N, C) | VX, Y € C^, u(gX, gY) = u{X, Y)} . 

Equivalently, Sp(n, C) is the subgroup of elements g £ GL(iV, C) subject to 
the relation t g Jg = J. 

From now on, G will denote the complex symplectic group defined above. 
Restricting the usual Cartan involution of GL(N, C) yields a Cartan involu- 
tion of G. As a consequence, K = U(iV) fl Sp(n, C) is a maximal compact 
subgroup of G, also called the compact symplectic group and denoted by 
Sp(n). 

1.2. Maximal parabolic subgroup of Heisenberg type. Let us recall 
some facts regarding complex Heisenberg groups. Let m = n — 1 and consider 

H 2m+i = |( S] j) eC x C 2m } 
equipped with the product 

(s, X)(s', X') = + s' + ±u{X, X'),X + X'^j , 

where CO = U)fn 

denotes the complex symplectic form on C . 
The group G acts naturally on by linear applications, hence also on the 
complex projective space P iV ~ 1 C. The stabiliser in G of a point in P Ar_1 C 
is a maximal parabolic subgroup P with Langlands decomposition 

P = MAN ~ (C x .Sp(m,C)) x H 2m+1 . 
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Elements in the Cartan-stable Levi component L = MA are of the form 



a 














Sll 





S12 








a- 1 








S21 





S22 



€ Sp(m, C). 



1 











x 1 


Ira 








2s 




1 


- t x 1 


x 2 








Im 



l(a,S) 



with a € C x and S = Su Sl2 

The Lie algebra g of G then admits a Gelfand-Naimark decomposition 
= n + m + a + n and the analytic subgroup N of G with Lie algebra n is 
another copy of H^ m+1 which embeds in G via 

(1) ( S ,(X!,X 2 ))h 

X 2 

1.3. Degenerate principal series. For a G C x , we denote [a] = y^y. Let 
(/i, <5) € C x Z. Such a couple defines a character x^,<5 of P by 

X^, 5 (l(a,S)) = \ar[a] s . 
From now on, we assume fi to be purely imaginary, so that \^,8 is unitary. 
Definition 1. The induced representation 

tt^s = Indp Xv,S ® 1 
is called a degenerate principal series representation of G. 
These representations may be described in several ways. 

1.3.1. Induced picture. In this model, 7r Mj< 5 is realised on a space of square- 
integrable sections of the line bundle Gx X/±s C over the flag manifold G/P ~ 
pN-iQ ^ dense subspace of the carrying space in this picture is 



^ = {/eHC w \{0}) 



Va € C x , /(a-) 



-M-iVr 



'/}■ 



By homogeneity, functions in are determined by their restriction to the 
unit sphere in C^, which we shall always identify to the 2N — 1-dimensional 
Euclidean sphere S 2N ~ l . The space V^s is defined as the completion of V^° s 
with respect to the L 2 -norm on S 2JV_1 , and G acts by left multiplications. 

1.3.2. Compact picture. Besides considering sections over the flag manifold 
G/P, one may restrict the induced picture to sections over K/{LC\K) where 
L denotes the Levi component C x .Sp(m, C) of P. Considering the compact 
symplectic group as the orthogonal group of a quaternionic vector space gives 
the identification 



(2) 



i27V-l 



~ Sp(n)/Sp(m) 
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so that 

K/(L n K) ~ Sp(n)/U(l).Sp(m) ~ S 2Af ~7U(l) 
and TTuS is realised on 

L 2 {S 2N - 1 ) S = {feL 2 (S 2 ^ 1 ) | V0 € R , f(e w •) = e - i<w / } . 

The action of G in this picture is slightly more complicated than in the 
induced one. However, its restriction to K reduces to the regular action 
by left multiplication: vr^°J ipact (A;)(/) = f{k~ 1 -). More details about real, 
complex and quaternionic spheres and the isotypical decompositions of the 
associated L 2 -spaces will appear in Section [3j in order to analyse the K- types 
of the representations ir„ § and determine the behaviour of the Knapp-Stein 
intertwiners on these K- types. 

1.3.3. Non-compact picture. Another standard picture for principal series 
representations is obtained by restricting functions in the induced picture to 
N . More precisely, through the embedding ([I]), any / in V^° s gives a function 

on H 2m+1 defined by 

(s,X 1 ,X 2 )^ f(l,2s,X 1 ,X 2 ) 

and still denoted by /. It follows that tt^^ is realised in L 2 (H^ m+1 ). 

In Section [H we introduce a new model for degenerate principal series 
representations and discuss its advantages in their study. 

2. Fourier transforms and Knapp-Stein integrals 

Various integral transforms will be used in relation to Knapp-Stein in- 
tertwining integrals. We shall define them on the space ^(C^) of rapidly 
decreasing functions and extend them to the Schwartz space tS^C^) of tem- 
pered distributions by duality. The complex Fourier transform of / € S(C N ) 
is defined by 

F C »f(0= I f{X)e- 2 ™^^ dX. 
Jc N 

The factors in the product decomposition of will be labelled C™ with 
i € {1,2} so that = C™ x C2 and the partial Fourier transform with 
respect to the i-th variable will be denoted by J-Qn. Thus for / 6 ^(C^), 

•Fcs/(*i,&)= / f(X l ,X 2 )e- 2mRe ^^dX 2 . 
Finally, the complex symplectic Fourier transform is defined on S(C N ) by 
F S ym P f(0= f f(X)e- 2 ™ Rc ^dX, 

Jc N 

that is 

The following lemmas state elementary properties of the above transforms, 
to be used further. 
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Lemma 1. Let f € V-^s and a € C x . Then V(Xi,£ 2 ) G C w 

^/(aX^o- 1 ^) = |ana] 5 Jb ? /(^i,6) 
Proof. By definition, 

^q/^i.a- 1 ^) = I f(aX 1 ,X 2 )e~ 2 ™ Re ^ a ~ 1 ^dX 2 

= \a\ 2n [ f(aX l ,aX 2 )e- 2i7rRe ^ X2 '^ dX 2 

= \a\ 2n \ar 2n [a] 5 ^f(X 1 ,^ 2 ), 
hence the result. □ 
Lemma 2. Let f G S(C N ). Then V(«,v) € C? x C£ ~ C^, 

o j; ymp o p-^j f( u , v) = f(v, u). 

Proof. Let us compute: 

f(u,v) 

= [ f ^iyiy-2i*K C {(v',v)Hv\u)-{u",vi)-(v">,v")) dv »> du » dv » dv > 

= f f(u", v "') e -^^((v',v-u")+(v",u-v'")) dv >» du » dv » dv > 

./q?xq i xC£xC£ 

f(u", v"')5 (v - u") 5 (u - v'") dv'" du", 

hence the expected equality. □ 

Let us introduce some more notations: e denote the real matrix of size 
2N defined by blocks as follows: 



[ In 








-In \ 



and if / is a function on R 2N , we denote by f £ the function X i — > f(sX). 

Remark 1. Under the identification between x R^ and given by 
(Xi,X 2 ) i — > Xi + iX 2 , the transformation e induces the complex conju- 
gation. It follows that the complex Fourier transform can be seen as the 
transform J- e defined on <S(R 2W ) by: 



f(0= [ f(X)e- 2i ^ x ^-^^d(X 1 ,X 2 ), 

JR?xR% 



so that F £ f = {F^2N f) £ . Indeed, if £ = (£i, £2) and £ = £1 + ^2, it is clear 
that 
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Definition 2. If p is a function in C^°(S 2N ~ 1 ) and A is a complex number, 
we denote p\ the function defined by extending p to M. 2N \ {0} by 

Px (rX) = r x p(X) 

for r > and X G S 2JV-1 . 

Following p], we also consider the meromorphic function in the complex 
variable A defined by 



\-N--k 



B 2N (\,k) =TT- A -"i 



r(AT + MA) 

p fk-X\ 



2 ) 

Then, denoting n k (R 2N ) the space of harmonic homogeneous polynomials 
of degree k over M. 2N , the following holds: 

Lemma 3. -Le£ p be the restriction to S 2Af_1 of a polynomial in 'H k (M. 2N ). 
The following identity between distributions on M? N depending meromorphi- 
cally on A holds: 

(t) FeP\ = B 2N {Kk)p e _x-2N- 

Proof. Following the lines of the proof of [21 Lemma 2.7], it is enough to 
prove that 

(3) (FeP\,gq) = B2n{\1)(P-\-2Ni9Q) 

for g G C^°(1R + ) and q G W l (M. 2N ) in the domain 

V 



-2N < Re A < - IN + 2 

to ensure that df| ) holds on 1R 2 ^. Local integrability of p\ and p £ .\ 2A' follows 
from the choice of the domain. Denoting by J^{y) the Bessel function of the 
first kind, the Bochner identity directly implies that 



g2JV-l 



so that 



T e gq{ruj) = / / g(s)q(oj')e- 2i7Trs{uJ '' £uj) s 2N - 1 dsda(oj') 
Jo Js 2 "- 1 

r+oo 

= 2irr l r 1 - N q(euj) / s N g(s)Ji +N ^ 1 {2nrs) ds. 
Jo 

It follows that 

/•+oo r 

(p\,Fe(gq)) = / r A p(w)-F £ 59(™)r 2Ar_1 drda(u) 



where we set 



g2JV-l \j 



I(r, s) dsj p(ijj)q{eijj) da(u>) dr 
I(r,s) = 27Ti- l r x+N s N g(s)J l+N ^(27rrs). 
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The proof of Claim 2.9 in [2] ensures that I belongs to L 1 (R + x R + ,drds) 
and that 



J(r, s) a?r = B 2 n{\ l)g{s)s 



-A-l 



As a consequence, 



(Px^eigq)) = / p(u)q(euj) da(u) / / I(r,s)drds 
Js 2 "- 1 Jo Jo 

r r+oo 

= B 2 n{\1) I p{uj)q(euj) da(u) \ g(s)s^ X ' 1 ds, 
Js™- 1 Jo 

which implies that 

(px^sigq)) = B 2N {\i){p~x-2N,gq £ ), 

thus proving j3]) and the lemma. □ 

Remark 2. The link between Fourier transforms and Knapp-Stein operators 
relies on the observation that J^ymp provides a unitary equivalence between 
tt^^S an d Tt-^-fr. Indeed, denoting f a = f(a-) for / G S(C N ) and a € C x , a 
single change of variables leads to 

(J~sympf) a = \a>\ -^symp (fa' 1 ) ■ 

If moreover / € V-n,s then, by linearity, 
that is 

Finally, for any g S G, 

^symp (TT-,,,-*^)/) (0 = / f{ X )e- 2 ^^X,i) dX 

Jc N 

and since oj is preserved by G, it follows that .7 r symp 7r -^,-<5(g) = 7T ^,s(9)J'symp- 

Next we introduce the Knapp-Stein operators which will be related to 
•^symp after normalisation. 

Definition 3. The operator 7^ >( 5 : V-^-s — > V^ t s obtained by meromorphic 
continuation with respect to /i of the integral 

%,sf(Y) = ! f(X) \Reu(X,Y)\-^ N [Reu(X, Y)]~ S da(X), 

where da is the Euclidean measure on the unit sphere of M. 2N , is called 
the Knapp-Stein operator, as introduced in [6], associated to the parameter 
(ju,<5) G C x Z. 

Remark 3. The kernel defining the operator in the above definition depends 
only on [i and the class of 5 in Z/2Z. 
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The operators T^s enjoy the intertwining property, yet as always in Knapp- 
Stein theory [6] [7], they are not unitary at first. However, normalisation may 
be obtained by using the symplectic Fourier transform. More precisely, let 

< r ( l- M -jV ' 



2^ +N - L 2 \ / if 5 is even 

r(^) 

Y ( 2-n-N 
-2i^ +N ^—) L if S is odd. 



Then the followings holds: 



Proposition 1. The normalised Knapp-Stein operator associated to (n,6) 
is defined by 

As a meromorphic extension in the complex variable [i, it satisfies 
(t) 7^,(5 = F symp |v_ M _ 5 . 

As a consequence, TuS yields a unitary equivalence between tt_»_5 andiTuS- 

Proof. Following [2, Prop. 2.13], we identify \ {0} to R* + x S 2 ^ 1 by 
using spherical coordinates (r, X). Then any function in s is of the 

form 

h„. N (rX) = r-»- N h(X) 
for some h G C°° (S 2N ~ l ) satisfying h{e ie X) = e m h{X) for any X € S^" 1 
and 9 € R, that is h is a C°° function in L 2 (S 2iV ~ 1 )_ 5 . The spherical 
component h being fixed, it is enough to prove 

for the parameter [x in a non-empty open domain. We shall work on the set 
defined by Re fi > —N. On this half-plane, hn—N is locally integrable and, 
defining h £ ^ N by 

h e , lt - N {rX) = e- 2nr h ll - N (rX), 

one has lim h £ „_jv = h u -N in S(C N ), so that 
e^0+ 

F iymphii—N — lim J~ iymp^£,/i- AT- 

£^0 + 



F iymp^e,(i- N (^^0 



e ~^r r ^ e -^rsKc^Y )h ^ x)da{x y^-L dr 

o Js 2 "- 1 

[ Fr ( r f i +N ' 1 ) (s Reu}{X, Y) - ie) h{X) da(X) 

J S 2N-1 \ ) 



10 



PIERRE CLARE 



where J 7 ^ denotes the usual Fourier transform on the real line. Then, using 
classical formulas relative to Fourier transforms of homogeneous distributions 
(see [3]), one has 

F symp h e ^- N (sY) = , > / (sReu(X,Y)-ie)-»- N h(X)da(X) 

and letting e — > + yields 

T symp h,. N (sY) = {fl + { J^ +N J (Reu(X,Y)-iO)-»- N h(X)da(X). 



Recall that 



r ■- 1 | cos I ] .r ' — / sin 



fir(n + N)\ ._ U _ N . , 
( -J |z| ^signfa) 



= ^ifOH-JV) [ l g l ^ * , jgj " ^signfo) 

\ v r(i±^)r(i^) r(^)r(^^ 

using Euler's formula. The duplication formula satisfied by T implies that: 

r(^ + iv) = 2^-^r^ + iV ^^ 1 + ^ + Ar 



Finally, we notice that h is even if h^-N belongs to V—u,~s with 5 G 22 
and odd otherwise, so that J r symp /i /i _Ar(sy) is equal to 

— JT^n\ s ~^~ N / \Reco(X,Y)r- N h(X)d<r(X) 

2 p ^ l-fj,-N ^ Js 2N - 1 
in the even case, while in the odd case it amounts to 



r (ggg 

-2i p f 2-jt-jV 



■a - ^ / |Rew(X,y)r^- Ar sign(Reo;(X,y))/i(X)da(X), 



thus proving ([J]). The last statement then follows from Remark [2j □ 



For future reference, the main Fourier transforms of use in what follows 
are listed below. 
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Summary of integral transforms 
Complex Fourier transform: 

Jc N 

Complex symplectic Fourier transform: ^"symp/CO = ^c N f(JO 

J>ymp/(0=/ f(X)e- 2 ™ Re «( X &dX 

Jc N 

Partial Fourier transform: on C N ~ C" x 0> 

^f(Xi,^)= [ f(X 1 ,X 2 )e- 2 ^ Rc ^^dX 2 
Real conjugate Fourier transform: F e f{£) = T^at/ 

^/(a,6) = ^ c -/(Ci+^2) 

Tef(0= I f(X)e- 2i « x ^-< x ^ d(X 1 ,X 2 ) 

3. Restriction to Sp(n) 

The determination of a i<C-type formula for 7r^ j( 5 relies on some known 
facts regarding the representation theory of orthogonal groups over R, C 
and H. More precisely it will involve the isotypical decompositions of square- 
integrable functions over the Euclidean unit sphere in IR 4n ~ C 2n ~ HP. 

3.1. Real, complex and quaternionic spherical harmonics. Let us fix 

the following classical identifications: 

(4) HP ~ (C n + jC n ) ~ ((R n + iR n ) + j {R n + iR n )) 

where i, j and k = ij denote the standard quaternion units. Then the unit 
spheres S\(-) of those isometric vector spaces all identify to S 2Ar_1 and carry 
compatible left actions of the corresponding orthogonal groups as follows: 

Sp(n) r> Si (HP) ^ Sp(l)~SU(2) 

n l\ U 

U(2n) rx Si(C 2n ) ^ U(l) 

(5) n i\ u 

0(4n) Si(]R 4n ) ^ {±1} 

g2AT-l 
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The right column displays the right actions of scalars of norm 1. As an 
identification between Sp(l) and SU(2), we fix the one given by 



(6) 



' i 




1 " 


, k i-> 


" i ' 


-i 


, 3 ^ 


-1 


i 





so that U(l) naturally appears as a Cartan subgroup of Sp(l) via the map 



(7) 







Square integrable functions on S 2N 1 decompose with respect to the cha- 
racters of {±1} as even and odd, while the component corresponding to 

5 G Z ~ U(l) is the space I? (S 2N ~ 1 ) S introduced in Section No 
ambiguity arises from the fact that the action was defined on the left there, 
since U(l) is abelian. 

Let us now turn to the decomposition of L 2 (S 2 ^ 1 ) into irreducible rep- 
resentations of 0(4n) and SU(2n), that is the classical theory of spherical 
harmonics on real and complex vector spaces. More details may be found in 
Section 2.1]. 

As in Section [2] we denote by 'H k (M 2N ) the vector space of harmonic ho- 
mogeneous polynomials on M. 2N of degree fcsN. It is also useful to consider 
the space T-L a, ^(C N ) of harmonic polynomials of the complex variable and 
its conjugate, homogeneous of degree a in Z G C N and of degree j3 in Z. 
Under the identifications of there is a natural isomorphism 



r>27V\ 



n a ' p (c s 

Restricting functions to the sphere provides a complete orthogonal basis, 
hence a discrete sum decomposition of L 2 (S 2Af_1 ) into irreducible compo- 
nents of the left actions of 0(4n) and U(2n) in ([5]), namely 



(8) l 2 (s 2 ^- 1 ) ~ YT n k (i 



,2N^ 



k>0 



S 2N- 



, * E © na '"^ 

k>0 a+/3=k 



Taking into account the right actions in ([5]), one can refine ([8]) as 



2 / c 2iV-l> 



L (S 



£ 2 (s^L - 



(9) 
(10) 

in the real case and 

(11) L 2 {S 2N - V 

in the complex case. 



'«G2N 

E * 



r>2N\ 



k/To2N\ 



fce2N+l 



, * E ^(c N ) 

f3-a=5 



g2JV-l 



g2JV-l 
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Remark 4. The above discussion of the isotypical decomposition of L 2 (S 2N ~ 
with respect to the action of U(2n) x U(l) (resp. 0(4n) x Z2) only involves 
the irreducible representations of U(2n) (resp. 0(4n)) because the commu- 
tativity of U(l) (resp. Z2) implies that these groups have 1-dimensional 
irreducible unitary representations, which is not the case of Sp(l). 

In order to proceed with the same analysis over quaternions and write 
down the analogue of ([8]) corresponding to the action of Sp(n) x Sp(l) on 
, some additional notations are needed. Following [5., Sections 5 

and 6], we denote by V n ' the unique unitary irreducible representation of 
Sp(ra) corresponding to the highest weight (I, I' ,0, ... , 0) where I and I' are 
integers satisfying I > I' > 0. Similarly, Vj 1 denotes the irreducible j + 1- 
dimensional representation of Sp(l) ~ SU(2). 

Since U(l) naturally embeds into SU(2) via ([7]), this representation de- 
composes according to the characters of the circle. More precisely, if C5 
denotes the space of the character z 1— > z s of U(l), for S € Z, then 

(12) V( ~ C 5 . 

\S\<3 

Now identifying S 2Ar_1 to Si (W 1 ), the isotypic decomposition of L 2 (S 2JV_1 ) 
with respect to action of Sp(n) x Sp(l) defined by the first line of ([5]) is given 
in [5] by: 

l>l'>0 

Together with (fT2l) . this decomposition gives 

(13) L 2 (S 2Ar - x )^ ^® VU'QCs 

under the action of Sp(n) x U(l). 

3.2. The branching law Sp(n, C) 4 Sp(ra). The above discussion leads to 
the determination of the branching law Sp(n, C) 4- Sp(n) of ir„g seen in the 
compact picture. 

Proposition 2 (i^-type formula). The restriction of tt„s to the maximal 
compact subgroup Sp(n) decomposes into irreducible components as follows: 

7T d ~ V® V 1 ' 1 ' 

l-l'>\8\ 
l-l'=S[2] 

Every K-type Vn' 1 occurs at most once in this decomposition. 

Proof. The discussion of Paragraph 11.3.21 shows how tt^s can be realised on 
1? (S 2 ^ -1 )^. In this picture, the restriction to K of the action coincides 
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with the natural representation of Sp(n) on functions over the unit sphere of 
M n . Subsequently, the first statement reduces to fixing 5 in (fl~3j) . The fact 
that this decomposition is multiplicity-free relies on the observation that 
although a summand Cg appears in V[- V for various values of I — V, these 
do not involve the same space Vn' 1 more than once. □ 

3.3. Action of the Knapp-Stein operators on the K-types. The last 
result of this section describes the behaviour of the Knapp-Stein intertwiners 
on each K-type. Proposition [T] proves that 7^,5 intertwines tt^j an d 
As a consequence of the formula in Proposition [21 these representations have 
the same K-types V n ' ■ Taking into account the right action of U(l) seen a 
Cartan subgroup of Sp(l) via ([5]) and the corresponding isotypic decompo- 
sition (|13p of L? (S 2Ar_1 ) , one is led to study the restriction 

T% : Vk V ®C S — > Vy ® C_ 5 



n n 

Vn' 1 ' ® Vl~ V Vn' 1 ' <8> V[- V 



In order to specifiy how 7^,<j restricts to an operator of Vn' 1 , it is neces- 

IV IV 

sary to fix an identification between V n ' <S> *Cs and V n ' ® C_,5. Using the 
isomorphism ([6]) between Sp(l) and SU(2). it is done by choosing a non- 
trivial element w in the Weyl group PU(SU(2) : U(l)) and letting it act by 
conjugation on SU(2). Since such an action inverts the elements in the torus 
U(l), it provides an isomorphism t w from Vn' 1 to itself, that exchanges the 



summands Cg and C_,5 appearing in ([12 
(14) U U 



v{- 1 ' ^ v\~ v 



-5 



The restricted intertwining operator under this identification will be de- 



noted by 7jJ>: 



rj4,V ,W rj4,l' 

V,<5 — 'u,5 



From now on, w will be the class modulo U(l) of 



1 
-1 



Since this 



matrix of SU(2) normalises U(l), it defines a Weyl element, hence fixes the 
above definition of T^' 1 " ■ 

Propositions. Let 5 / 0. For 1,1' G N such that I -1' > \S\ andl-V = 5[2], 
the restriction T l ^' w of the normalised Knapp-Stein intertwiner T^g acts on 

Vn' 1 as the scalar 



n + 



l+V+u 
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Proof. Let p € V n ' <8> C VV <S> . By compatibility of the isotypic 



decompositions ([8]) and (|13p . p can be seen as the restriction to S 2N 1 of a 
polynomial in U l+l '{R 2N ). In view of Proposition Q] and Lemma [3l the oper- 
ator Tu,s maps Ph-n to B2n{^ — N,l + 1')p- u -n{J£ •). Under the identifica- 
tions (j4|), applying Je to vectors in HP from the left amounts to multiplying 
them by the quaternionic unit j from the right. It follows that 

^f W P = b 2n(v -N,l + l')t w (p.j) 

where j € Sp(l) acts via V[~ l . Identifying Sp(l) to SU(2) by ([6]) again, 
we realise V^~ l as the classical representation of SU(2) on homogeneous 
polynomials of degree I — I' in two variables x and y, denoted by [x, y]. 



Since j and w are both represented by the matrix 

lw (P.j) = P-(-h) = (~1) 5 P, 

j4,V 
' n,S 



1 
-1 



one has 



hence the conclusion: Tj,'g' W p = (— 1) s B2n(ij> — N, I + l')p. □ 



3.4. Analysis of 7To.o- Let us discuss the elements of the dual space G 
obtained from the degenerate principal series, that is the equivalence classes 
of irreducible unitary subrepresentations of {iTi\,5 , K x Z}, It is proved in [4] 
that 7TjA,<5 is irreducible if (A, 5) 7^ (0, 0). Moreover, the Knapp-Stein operator 
7iA,<5 ( or its algebraic version T^g in the non-standard model) exhibits a 
unitary equivalence between ir^s and 7r_jA,-<5- It is also established in [4] 
that 710,0 splits into the direct sum of two irreducible subrepresentations. 

The following result describes this splitting in terms of eigenspaces of the 
Knapp-Stein operators and specifies the if-module structure of the sum- 
mands. 

Theorem 1. The representation ttq^q of G is reducible and decomposes as 

vr ,o - tt^o © t{qj 
where the irreducible summands tt^q are characterised by: 

(1) their K-type formula: 

V{? and vr+ ~ ^ V% ; 
l-l'=2[A] 2-Z'=0[4] 

(2) the classical Knapp-Stein intertwiners: tTq (resp. Kqq) is the eigenspace 
for the eigenvalue 1 (resp. —1) o/7q,o acting on Vq^. 

(3) the algebraic Knapp-Stein intertwiners: tTq (resp. tTqq) is the eigenspace 
for the eigenvalue 1 (resp. —1) of Tq : q acting on I? (C 2m+1 ). 

Proof. To establish (1) and (2), we proceed as in the proof of Proposition 
[3j except that no choice of a Weyl representative is required to identify 
V 1 ^ 1 ' ® C s to V\~ V ® C_,5 when 5 = 0. For p € Vn' 1 ' ® C C Vn' 1 ' V 1 ^ 1 ' one 
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has ToflP = B2n(—N, I + l')(p-j) where j acts by the representation Vj 1 of 
Sp(l). Identifying Sp(l) to SU(2) so that j is represented by ^ J .and 

Vl~ v to ¥- l '[x,y),it appears that j acts on the variables by (x, y) i— > (y, — x) 
so that a 0-weight vector £ is send to ( — £ by j. It follows that the 
intertwining operator acts on every if -type V n ' of 710,0 by 

B 2N (-N,l + l'){-l)^r. 

1+1' 



Since B 2N (-N,l + l') = (-1 



by definition, it follows that 



T , oP = (-l)- l 'p, 

hence the result. We postpone the proof of (3) to the next paragraph where 
the algebraic Knapp-Stein operators are defined and studied. □ 



1 2 3 4 5 



rl,l' 



Vn appears in 7Tq 



V n ' 1 appears in tt q 



FIGURE 1 . Repartition of the .fT-types of ttq q 



4. A NON-STANDARD MODEL AND INTERTWINING OPERATORS 

This section is devoted to the description of a new model of degenerate 
principal series, in which intertwining operators happen to take an algebraic 
form. 

4.1. Non-standard model. Let /x and 5 be as above. The non-compact 
picture described in Paragraph 11.3.31 allowed to realise tt^s on the Hilbert 
space 



L ( H 2-+i) ~ L ( C 



L 



) ' 
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Let J-cxC™ be the partial Fourier transform denned on L 2 (C x C™ x C™) by 
FcxC?f(r,X 1 ,&) = / /(t,X lj X 2 )e- 2i7rRe ^+^^»dX 2 dt. 



Definition 4. The non-standard model Uu,5 of 7r M .5 is the image of the non- 
compact picture L 2 (H 2 : m+1 ) by 

: L 2 (C 2m+1 ) — ► L 2 (C 2m+1 ) , 

that is Vi^.s = L 2 (C 2m+1 ) as a Hilbert space and the action of an element 
g G G on is given by JbxC™ ^(sO -^cxc^- 

The equivalences between the induced, non-compact and non-standard 
models of 7T„ 5 are summed up in the following diagram: 

(15) a^s : V^s ^ L 2 (H 2m+1 ) — — > W Mj(5 

f"> f"i restrict. ^ J CxC m " 

/ 1 F 1 "-^H 

where, according to the embedding ([1]), 

F(t,X u X 2 ) = f(l,X 1 ,2t,X 2 ) 
for t G C and (Xi, JC 2 ) G Cf x Cg*. 
Lemma 4. Lei / G V^s- With notations as above, 



Proof. According to the notations in the definition (|15p of ot^s, one has 
H(t,X u £ 2 ) = [ F(t,X 1 ,X 2 )e- 2i7TRc ^ X2 ^dX 2 dt 

= [ /(l,X 1 ,2t,X 2 )e- 2 ^ Re<( *' X2) ' (r ' 6)) d(t,X 2 ) 

\ [ /(i,x 1) t,x 2 ) e - 2 - Re ^)-(i^))d(t,x 2 ), 



2 

hence the result. □ 



4.2. Algebraic Knapp-Stein intertwiners. We now come to the main 
point of this section, id est the proof that the normalised Knapp-Stein op- 
erators considered in Section [2] take an algebraic form once expressed in 
the non-standard model of the previous paragraph. More precisely, for 
H G L 2 (C 2m+1 ) , we let 

T^ 5 H(s,X l ,X 2 )= i ^[s^hIs.-X^-Xx 
I \ s 2 
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The jacobian determinant of the transform 

(s,X 1 ,X 2 )^ ( S ,-X 2 , S -X 1 
\ s I 

is easily seen to have modulus 1, so that T^g is an endomorphism of L 2 (C 2m+1 ) . 
which turns out to be the realisation of the normalised Knapp-Stein inter- 
twiner 7^ j( 5 in the non-standard picture: 

Theorem 2. For any 5) S ilx Z, the following diagram is commutative: 



V- 



1-i.S 



Proof. Let / G V^ _ s . Then 



L fi,S 



Ct/j.,6 ° lfi,6 JC~,u\l,l;2) = •TCxC^-' symp 



7^,5/(^,^1,^2) = ft 

1 

2 
1 

2 

1 

2 

T 

2 

T 

2 



r 



•^C^symp / ( 



2 




"2" 


5 / 










T 




r 



, 2 ,?2 



2 
r 



r r 
2' 2 



Xi 



[r 



^ —J~C n f ( 1, — £2, — , — -X] 



M \t]~ 5 h(t,-&,~X 1 



by Proposition [JJ 
by Lemma HI 

by Lemma [21 
by Lemma [Tj 

by Lemma HI 

□ 



thus proving that a^ s o T^s = T^ s a-^-S 

We can now complete the proof of Theorem [TJ Since Tq Q = Id^2(c2m+i), 
every function H 6 L? (C 2m+1 ) can be written in a unique way as H = 
H + + H- with Tq$H + = H + and Tq^H- = —H-. Indeed, 

H ± = ^(H± T , H) , 

which gives the expected characterisation (3) of tt^ in Theorem [TJ 

4.3. Perspectives. The existence of a non-standard model for degenerate 
principal series of Sp(n, R) and Sp(n, C) in which the Knapp-Stein intertwin- 
ers take an algebraic form relies on the special form of the nilradical of the 
inducing parabolic subgroup. It is then natural to ask if the same occurs 
with parabolic subgroups of Heisenberg type in other groups. 
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